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ABSTBACT 

^ The effects of test content reflecting black or white 
culture r on the perforiance of college-bound students of both races 
vas exaiined« Eight yrrf lire culture- specific iteiis vere developed for 
use in the English Osage euid Social Studies BeadiAg subtests of *the 
ACT (Aaerican College Testing) Assessient Prcgrai. Itesil vere * 
adiinistered -as four separate tests to equal nuubers of black and. 
white exauinees^ in several southern states. An analysis of. variance 
showed no significant interactions between test content and 
perforiance by race. Further, no systesetic differences in lean itei 
discriiination in4ices, lean nuaber of ites oiissions, or intenlal 
consistency coefficients were found. Collectively, these results 
Suggest that test coiitent did not have a aa jor effect .on any of the 
variables studied. Suggestions for future research ^^and the 
isplica^iohs of the use of cultural content in both, classrooa and 
standardised tests are discussed. (Author/CP) 
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^. ABSTRACT 

PERJ'ORMANCE OF BLACK AND WHIT£ STUDENTS 
• ON TEST MATERIALS CONTAINING CONTENT BASED ON BLACK AND , ^ 

WHITE CULTURES V - * 

'J 
% 

The purJxDse of this study was to examine the effects on the performances of / 
black and white examinees of test content based on black and Vhite cultures.'' 
Test materials were special'units developed for use. in subtests of the ACT 
Assessment Program. The test materials were administered to both black ajid^ 
white examinees in several southern states. No significant interactflons be-"^ 
tween test content and examinee performance by race were found. Furyther, no ^ 
systematic diff>erences in mean item discrimination indices, mean^a^upjaer of ^ 
item omissions, or internal consistency (KK^^ coefficients were found/ Col- 
lectively, these results s\aggest that test content did no^: have a ma jox effect 
bn any of thfe vari^les studied.' - . ' ' • ^^-^^ \ 
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^c^- * Simultaneous with increased access to higher education of members of 
, minority groups traditionally, denied such access has come an increased con- 
. cern- with the impact of achievement testing *on the decisions made by colleges 
' using' test results. In peirticular, much attention is being given to the 
t issue of bias in achievement testing and its impafct on those who are tested. 

In general, attention has focused on test bias from one of two perspectives 

. ^ . . . f 

bias in a test v^en.a criterion is available and bias that may exist in a 
test when no criterion exists or is relevant, ^ . 

' . . • - " . <• . 

The first perspective has focused on test bias when one or more 
cjriteria (such as grades for selection purposes) are available. If the 
iQccess of the members in a particular subgroup is predicted equally well 
/ using the prediction procedure detendined appropriate for the entire group, 

then the instrument is not considered biased (^lofndike/ 1971; 'Darlington, - 
1971; Cole, 1973). However, if a particular • test is -predictive of suecess 
as the educational process now exists, then the issue of bias is not neces- 
sarily dismissed. Bias may exi-st in the criterion measures used (such as 
^ , grades) which are reflective of the curriculum and the educational process 

which may, in fact, be equal even within an .institution for various subgroups. 

The second perspective has focused on bSas a^ it exists in the test 
itself, without relating examinee performance tcJs^ny external criterion. ^ 
Several studies (Cardall and-Coffm^ 1964; Cleairy and Hilton, 1968; ^goff 
and Ford, 1^71;" Eagle and Harris, 1969) have defined test bias as -a statis- 
tically significant items-by-i:ace interaction in an analysis of variauice procedure. 
If a significant interaction is found, ope of several procedures can be used to 
identify which items are differentially affecting the .performance of the racial 
subgroups under study. However, the identification o^ such biased items does not 
necessarily prove that the test as a whole is biased.^ Tl^e items identified as 
having dif ferent^ial effects on two subgroups may be balanced for the two sub- 
groups thereby indicating that the test does not favor one particular subgroup 
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oyer ^^riother but that the test as a whole produces test scores that treat 

each subgroup in an equitable manner* 

If however, the items that cire identified as biased by this procedure 

axe not bal^anced, then a' more Intense analysis of these items must occur to^ 

determine why the items function differently for efferent subgroups. 

Several possibilities have- iJ^en suggested by Eagle and Harris: 

"Though this study strongly suggests the operation of sig- 
nificant ^Test X Culture -interaction effects, the specific 
test chciracteristics (consent ,\ cognitive function, technical, 
features such as speed, etc.) and specific socio-cultural , ^ 

characteijistics (ethnicity, economic classic attitude, mental 
ability level) which may be entering into tVifese interactions, \ 
'have not been examined. V • ' ^ ' 

One of the most likely explanaticins of why iten^s- might favor one 
stabgroup is test content. That' is, test content reflecting the majority 
culture may favor members of that peirticulcir subgroup. Conversely, test content 
reflecting the minority culture may favor minority slbg roup members and may 
be detrimental to majority subgroup members. 

The purpose of this studV'is to examine the effect of test content on 
black an^ white student performance. To do this, test materials were 
developed, reflecting both black and white cultures, for two subtests of i 
the ACT Assessment excunination. These tests wei^e then administered to 
both black and white examinees. 

Delta resulting from this study will then, be used to address the follbwing 
questions: 

1) Are the performances of black and white examinees differentially 
affected by black and white English content? By black and white 
social studies content? 

2) ^Does the performance of black and white excuninees differ within a 

subject cirea? That is, does one subgroup significantly outscore 
ft 

another? 




3) Does the average difficulty level of the black and white tests 

differ within a subject area? ^That is, is one test significantly 

\ • 

hardfer for the 'two. subgroups than the other? * - 

4) Do the mean item discrimination, item* omission indices, an^ 

-reliability coefficients differ for black and white examinees? 
• • • 

Collectively, these "questions will examine the impact of test content 
on examinee performanc^ black^and white examinee performance, eind on the 
test as a whole. 

mi 

INSTRUMENTS ^ • 

The test materials used in this study were developed for inclusion in 
the ACT English Usage Test and the Social Studies ^Reading Test, Two' English 
Usage tests were developed: one that pertained to black cultvire (18 items) 
cuid - one . that pertained to white culture (17 items) • The test based on content 
identified with the white culture included such topics as motorcycling, U,S, 
fight for independence, and English literature* ^e second English test, 
v^ich was written from a^'black viewpoint, was based on a black woman's 
reflections of her Civil War experiences. Both units were developed to , 
measure the same standard English usage and mecheinics skills Though these 
skills might be consjidered a 'source of bias, they are representative of those ^ 
usage skills taught nationally in high spools and colleges today. A more 
coAiprehensive review of the emphases in high schools and college leinguage 
arts curricula is pjcovkded in a monograph wi;itten 'by Huntley (1977) . Thus# 
the units were deve'ioped so that only thie content of the xinits, not the skills 
measured by the items, differed between the two units. / • 

Three social studies tests were *also developedi for this study. Two of 
these uiilts, which were written by black item writers^ pertained to black * 
culture. 5Iach test contained a passage wHlch was based on the history of blacks 
in America, including theij: struggle for their legal and civil rights in the / 

nineteenth and twentieth centuries. The first black test (B^) had 20 items; y 

) 



the second black test (B ) had 21. The third social studies test (W^) was 

; • ■ ■ - - 

based on white content and pertained to seventeenth century English history. 

V • \ ■ 

This unit contained 20 items. Again, all three tests were developed to 

■ ' ■ ■ • . / 

measure the same cognitive skills. Both the black and white English tests 

were combined and administered as a single test to the same group of 
examinees; the three social studies tests were administered as individual 
tests to three different groups. 

A summary of tlje tests used in the study is given below in Table 1: 
Also* included are the numbers of items contained in each test. 
' ^ ' Table 1 

Description of Units 

t 

^ Subject of Unit 



Culture Reflected 
in Unit 



White 
Black 



English Usage 

17 items 

18 items 



Social Studies 



20 items (W^) 



items (B^) 



21 items (B^) 



SAMPLE 



* The four tests were administered to examinees at selected test centers in ten 
locations (all in southern states). About half of the examinees in' each of 
these locations were black^ and about^half • were whxte. Each examinee dt a 
particular t^st sitestook only one of the four tests, and the tests were ran- 
domly distributed to the sample and arranged ^o that each 'test was completed 
by some examinees in every testing' site . The tests were administered. under . 
' standardized" conditions, which allotted twenty minutes for the ac^jninistra'tion 
of the test9-. One hundred thirty-three (133) black and 133 white students 

took both the" English tests,- 136 of each group took test B^; 140 of each race . 

/ ' - . ' ■ 

topk B ; and 126 took W . | 

2 ' J- #-»- 



ANALYSIS 

The effect of black and white -English content on black and white 

jBxaminee test performance was determined using a Type I analysis of variance 

(Lindqu4.st, 1953). The analysis for social i^tudies content was a two-factor 

analysis of variance* The two factors in both analyses were test content and 

race. Test performance was computed as a proportion of the number of items 

answered correctly to the number of items in the test since the number off 

items was not constant across tests. Discrimination indices wei:e ccfiputed 

for each item using the common D-index (Ebel, 1972) • The 27 percent extreme 

groups were defined using the total scores^ of the ex^iminees on the corresponding 

ACT national tests (English Usage Test and the Social Studies Reading Test) • 

t 

The number of omissions was calculated as the average number of omits on the 
last five items of the test. Internal ccHisistency (KR20^ reliability co- 
efficients were also calculated. 

Results 

Th^' results of the anai^yses of the effect of black and white test content* 
on peprformance byVrace are shown in TcQ^les 2 cind 3. In Table 2, the mean 
test performances of bcthjbl^k and white exciminees are shown for the Engl^h 
tests and for both sets of social studies tests. In Table, 3, the analysis 

4 * 

of variauice tdbles are .presented for all three analyses. 

The lack of a significant interaction between test content and perfiprmance 

by race for the two English tests suggests that black and whi^e students did 

not respond differentially to the black arvi white English tests. The 

f - ' 

significant content main effect seems reasonable since the black English 
r 

test was easier than the white English test for both black and white examinees. 
In addition) the white excuninees scored higher thain the black examinees on 
both te^ts; consequently, the main ef fecit for race was significant. 
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' " Table 2 



' f 
d 


4 % 

( 

1 


Black and White Exciminee 
Mean Test Performance 




Cel-l Mean Proportions 




Test Content 


Black £x££miQees 

f 


White Examinees 


Coiribined Races 




English Tests 






- 




Black Content " 


44.11 , ^ 


57.56 


,50.84 




White Content 


31.76 


41.84 ■ 


36.80 




Combi-^ned Tests 


37.94 


49.70 


^' 43. 82"* 


II. 


Social Studies 
Tests and 




* 




^ 


Black Content 
White Content 


' 40.66 
42.94 


50.52 
' 52.14 


45.59 

47.54 

46.56 , 




Corhbined Tests 


41.80 


51,33 


III- 


Social Studies 
Tests B^ and 










Black Contejit 


35.71 


42,55. 


39,13 




V/hite Content 


42.94 


52,14 


47.54 




Combined Tests 


39.32 


■ 47.35 ■ 


143.^3 
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' TABLE 3 ' 




•• 
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I., English Usage Test6 

^ ____ 


ANALYSIS, of VARIANCE SUMMARI^ TABLES 


1-> — 


• 


.Source ^ 


• df 


MS 1 




F 


Betweeo Subiects 
Race 

Subjects within race 


< 

264 ^ 


18416. 04r 
395.131 




46.607 *' 


Within Subjects \ 
^ Content 
* Ha'pg V Content 
Content x Subjects 
within race 


• •• 1 
1 

264 

/ 


26208^188 
376.778 

149.097 
• » 




■ 175.780,*-- 
2.528 . • ■ 



II., Social Studies Tests and 



Source 






MS 


F 


Content 


1 . 




498.149 


1.379 


Race 


1 




•11927.480 


33.03Q^ 


Contentj^x Race 
Within Cells 


• , 1 
520 

t 




13.670 

O ^ T TIT 

361.111 


.038 


I. Social Studies 


Tests and W^ 




« 


J 


ScLrce 


df 




MS 


^ F - 


Content 






9376.410 


^9.306* 


Race 


i 




8426. Z77 


"26.336* 


Content^ x Race 


1 ■ 




186.073 


0.582 


Within Cells 


528 




319.954 


* 


*p/.001 


' 10 


L 


A 


r 



f - 
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•In the analysis-^ of variance for social studies te^ts B and W ,.the 
interaction of content and .race was also nonsignificant ♦ The test of the 
main effect of content was also not significant, indicating that ^le two tests 
Were not significantly different in difficulty for the combined races* Ohe 
test of the main effect for race was significant with white examinees scoring 
'higher on both tests* 

In the analysis for the eff4::t of social studies tests ah'd on 

performance by race, the interaction was agcdn nonsig^if iccunt. Botl> tests^ 

oT'the main effects w6re significant, indicating that although whites outsocred 

blacks on both tests, both bla^cks and whites scored higher on the white test* 

• 

Disci^imination ^ \ 

N 

The mean item discrimina'tion valjaes computed for both the English tests, , 

test B and*W , ,and tests B' and W were higher for the v^ite examinees on 
11 % 

all tests* In addition, the indices- computed for the white examinees tended 
to be higher for the black content than for tbe' v^ite cont^t* Ihere wei^ 
no consistent effects for the discrimination values computed for items taken 
by the\bl^ack examinees* These results seem to indicate that: 1) higher and 
lower ability white, examinees, were consistently differentiated jnore effectively 
than were higher and .lower ability black examinees .regardless of t^st content, 
and 2) the black tests differentiated h^her aind Aower ability white examinees 
better than the white tests* ' . . 

Omissions - 

The mean number of omissions can be used as a gross indicator of th6 
speededness of a test* That is, th^ greater the mean number of omissions, the 
more speeded the test* One might expect that since-"white examines typically 
outScore black examinees, white examinees may tend ^ to omit fewer items than 
black examinees, regcirdless of test content* However, this, was not the case in 



this study. The mean number of omissions computed for both races on English 

11 



test content indicate that whit^ examirffees tended to omit mor^ items on 
the black test than did black examinees and that black examinees tended to 
omit mpre items on the white test than did white examinees ♦ 

\ ■ •• • 

In social studies tests B^,and , the mean numbei* of omissions for the 
b'lack exaSInees-was greater for both tests, with blacks omitting more items 
or) the black test than on the white test. 

s 

• similarly r in social studies tests and W^, the mean number 'of omiss^ions 
for the black examinees <^as*gr§k^er for both tests. In these tests, both 
blacks and ,whites tended to omit mo^re items in the black test than in the white - 
test* Ihis same result was found in the compatrison of tests and W . 

Reliability ' ^ 

r The KR coefficients computed for both black and white tests on the , 

basis of the black and white examinees were tested for population" equality 
* *- ^ • ^ t « • . 

using a procedure described by Feldt (1969). In none of the comparisons did 
the reliability coefficients based on black and white examinees on the same 
test differ Significantly. However, 'the coefficient computed ♦for black exam- 
inees on the black English test wa^ sligljtly higher than it was for the white 
examinees on the same test; conversely, the KR^q value wais higher for white 
examinees on the white English test than it was for black exaininees on the 
6ame test. ^ In all ,the social studies Comparisons, the KR20 ^^^^^^^^^^^^ com- 
puted on the white samples were higher .than* those for the black- sairples, 

DISCUSSION 

Although white examinees scojred higheij than black examipees regeurdless 
of test content) the lack of significcint interactions between test content and 
test performance by ra<re in either Engl4^1\ or" social studies suggests that 
bl^ck cuid vfhite^ content does npt differentially affect black and v^te test 
perfonR2mce. These Results are in contrast to those found by. Medley and Quirk 
(1974). In thCir study, the performances of black 4uid white examinees on 



black, modern, and tra(Jitional test items contained on the general culture 
section of the National Teacher Examination were examined, Tflie results of 
their study indicated* that blacks outscored whites on black content, How- 
•ever, in the current study, no consistent effects were found for t>lack and 
white test forms in either English or social studies,, ^fliat is, the black ^ 
content (or white content) does jiot seem to be systematically easier or harder 
for either group 'in English' or social studies. It is possible that this 
^result is pcirticulcir to a given test rather than directly related to the test's 
content, ; 

Examination of the mean item discrimination values indicated that higher 
and lower ability white examinee's are differentiated more effectively than 
black ^examinees regardless of test content and that ^ite examinees are differ- 
entiated better with tests of black content than white content. No' systematic 
differences in discrimination indices were found for black examinees on eithey 
test, Collectiv^ly, 'these results suggest .that no systematic reliable effect 
o€ \^st. content on examinee tjperformance was reflected through this test char- , 
acteristic, , - ' * 

The results for the mean number of item omis§ion|f for the English and social 
studies tests tended to differ, as expected. In fenglish, whites and blacks tended 
to omit items more frequently -in the tests containing content of the other race. 

In social- studies, black examinees omitted more items in both the black cind white 

( . " . ^' ' , 

tests than did white examinees with the most omissions occurring on the black 

test. On the basis of these results, the effect of omissions and speededness, 

to €ui exter^t*^is unclear; Ijowever, it is poss4.ble that the effect could be 

significemtly related t6 the particulair subject area (English pr social studies) . 

Further research should Se aimed at clarifying this effect. 
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though the KR^QCoef ficients tended to be higher for white examinees 



. than ttey were for black examinees on the same test, the differences were within 

A 
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the range attributable to sampling errors at p=,05. Thus, the tests seem ^ 
to be cdx)Ut equally internally consistent for both samples, 
IMPLICATIONS * > ^ ' " ' 

Although this study was designed only as a preliminary investigation 
'of the effect of test content on black and white student performance, the 
results obtained in the two major subject areas did not indicate a differential 
ef f ect ,of content on examinee ^performance by race* The results,, however , \ 
did support those trends found by Dreger and Miller (1960, 1968) and Shuey 
(1966) that white examinees tend to outscore black examinees. 

The results of this study should i>e tempered by the fact that only^ 
liihited samples of behavior in each content domain 'were obtained from the 
examinees^ It seems logical, therefore, that future research on the effect 
of test content on test pe^ormance be aimed at obtaining a number of comparisons 
within each subject area, €o that general tr^ds across sxibject areas can -be 
identified • If test content does indeed tehd to re^sult in one race receiving 
a consistent advantage at the expense of another, then both classroom and , 
st£Uidardized test developers Should try to adjust or compensate for this type 
of test bias by balancing or eliminating biased content in their tests. If 
test content does not seem to differentially aff6ct black and white test 
performance, and yet inconsistent differences still occur between the per- 
formpince of the two races on a test, then other potential vari^Jbles of bias 
should be closely examined^ It is likely that such significant differences in , 
test performanbe, could be influenced «^ 1) other test characteristics (technical 
features, such as speededness) ; 2) item characteristics (cognitive skills \ 
measured, wording, types of Alternatives); or 3) specific sociocultural 
characteristics (economic class, mental ability level) which were not 

V 

controlled for in this study. Such sources of bias should be considered 
from these three perspectives. In any Case , if test content does not seem 
to differentially affect test perfomance of the two races, then the use of 
diverse cultural content in both classroom and standardized tests should 

ii : : 
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probably become more widespread. Given the significant number of individuals 
of different cultural background who are seeking access to higher education, 
there is much to command the inclusi'on of content materials reflecting 
.diverse cultural backgrounds if such content does not have an^ adverse effect 
iisi any cultural group. , ' . 
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